An experimental paradigm for studying sense of agency in joint human-machine motor actions.

In this paper, we propose an experimental technique for studying the sense of agency (SoA) in joint human-machine actions. This technique is based on the use of an electromechanical finger-lifting device that enables a joint motor action initiated by a participant and completed by the machine. The joint action, later referred to as an "active-passive" action, was implemented as a reaction time task and contrasted with other levels of participant's involvement, including active movement, passive movement, and observation of a dummy's movement. In each trial, a feedback sound signal informed the participant whether they had performed the task successfully, i.e. faster than a threshold, which was individually adjusted in the beginning of the experiment. In the active condition, the result depended on the participant, while in other conditions it was preprogrammed for the servo. In context of this task, we studied direct time estimates made by participants and auditory event-related potentials (ERP) in 20 healthy volunteers. The amplitude of the auditory N1 component in the responses to the feedback sound showed no significant effect of activity and success factors, while its latency was shorter in successful trials. Interaction of activity and success factors was significant for subjective time estimates. Surprisingly, the intentional binding effect (subjective compression of time intervals, which is known as a correlate of SoA) only emerged in trials of active condition with negative results. This observation was in contrast with the fact that the active and active-passive movements were both voluntarily initiated by the participant. We believe that studying SoA with the proposed technique may not only add to the understanding of agency but also provide practically relevant results for the development of human-machine systems such as exoskeletons.

A passive BCI for monitoring the intentionality of the gaze-based moving object selection.

Objective.The use of an electroencephalogram (EEG) anticipation-related component, the expectancy wave (E-wave), in brain-machine interaction was proposed more than 50 years ago. This possibility was not explored for decades, but recently it was shown that voluntary attempts to select items using eye fixations, but not spontaneous eye fixations, are accompanied by the E-wave. Thus, the use of the E-wave detection was proposed for the enhancement of gaze interaction technology, which has a strong need for a mean to decide if a gaze behavior is voluntary or not. Here, we attempted at estimating whether this approach can be used in the context of moving object selection through smooth pursuit eye movements.Approach.Eighteen participants selected, one by one, items which moved on a computer screen, by gazing at them. In separate runs, the participants performed tasks not related to voluntary selection but also provoking smooth pursuit. A low-cost consumer-grade eye tracker was used for item selection.Main results.A component resembling the E-wave was found in the averaged EEG segments time-locked to voluntary selection events of every participant. Linear discriminant analysis with shrinkage regularization classified the intentional and spontaneous smooth pursuit eye movements, using single-trial 300 ms long EEG segments, significantly above chance in eight participants. When the classifier output was averaged over ten subsequent data segments, median group ROC AUC of 0.75 was achieved.Significance.The results suggest the possible usefulness of the E-wave detection in the gaze-based selection of moving items, e.g. in video games. This technique might be more effective when trial data can be averaged, thus it could be considered for use in passive interfaces, for example, in estimating the degree of the user's involvement during gaze-based interaction.

Size matters: How reaching and vergence movements are influenced by the familiar size of stereoscopically presented objects.

The knowledge about the usual size of objects-familiar size-is known to be a taken into account for distance perception. The influence of familiar size on action programming is less clear and has not yet been tested with regard to vergence eye movements. In two experiments, we stereoscopically presented everyday objects, such as a credit card or a package of paper tissues, and varied the distance as specified by binocular disparity and the distance as specified by familiar size. Participants had to fixate the shown object and subsequently reach towards it either with open or with closed eyes. When binocular disparity and familiar size were in conflict, reaching movements revealed a combination of the two depth cues with individually different weights. The influence of familiar size was larger when no visual feedback was available during the reaching movement. Vergence movements closely followed binocular disparity and were largely unaffected by familiar size. In sum, the results suggest that in this experimental setting familiar size is taken into account for programming and executing reaching movements while vergence movements are primarily based on binocular disparity.

Consciousness in a multilevel architecture: What causes the lateralization of effective connectivity under resting state?

Here we present our answers to a critical commentary by Elkhonon Goldberg on our recent publication (Velichkovsky et al., 2018). To avoid discussions about novelty effects in the human brain activity and memory processes, we narrowed down this response to a reanalysis of our data along the lines proposed in the commentary, namely to comparing the effective links between symmetrical brain structures during the first and the last parts of a prolonged resting-state fMRI experiment. We also tested for sex differences in our results and checked for a stability of top-down interactions during the course of experiment because learning is often expressed in the weakening of upper level control over low-level mechanisms. Our attempts to test the predictions based on the novelty hypothesis has led to mixed results suggesting that the discovered right-to-left dominance of causal connections at rest may have a deeper origin than supposed in the Goldberg's commentary.

Consciousness in a multilevel architecture: Evidence from the right side of the brain.

By taking into account Bruce Bridgeman's interest in an evolutionary framing of human cognition, we examine effective (cause-and-effect) connectivity among cortical structures related to different parts of the triune phylogenetic stratification: archicortex, paleocortex and neocortex. Using resting-state functional magnetic resonance imaging data from 25 healthy subjects and spectral Dynamic Causal Modeling, we report interactions among 10 symmetrical left and right brain areas. Our results testify to general rightward and top-down biases in excitatory interactions of these structures during resting state, when self-related contemplation prevails over more objectified conceptual thinking. The right hippocampus is the only structure that shows bottom-up excitatory influences extending to the frontopolar cortex. The right ventrolateral cortex also plays a prominent role as it interacts with the majority of nodes within and between evolutionary distinct brain subdivisions. These results suggest the existence of several levels of cognitive-affective organization in the human brain and their profound lateralization.

Gaze3DFix: Detecting 3D fixations with an ellipsoidal bounding volume.

Nowadays, the use of eyetracking to determine 2-D gaze positions is common practice, and several approaches to the detection of 2-D fixations exist, but ready-to-use algorithms to determine eye movements in three dimensions are still missing. Here we present a dispersion-based algorithm with an ellipsoidal bounding volume that estimates 3D fixations. Therefore, 3D gaze points are obtained using a vector-based approach and are further processed with our algorithm. To evaluate the accuracy of our method, we performed experimental studies with real and virtual stimuli. We obtained good congruence between stimulus position and both the 3D gaze points and the 3D fixation locations within the tested range of 200-600 mm. The mean deviation of the 3D fixations from the stimulus positions was 17 mm for the real as well as for the virtual stimuli, with larger variances at increasing stimulus distances. The described algorithms are implemented in two dynamic linked libraries (Gaze3D.dll and Fixation3D.dll), and we provide a graphical user interface (Gaze3DFixGUI.exe) that is designed for importing 2-D binocular eyetracking data and calculating both 3D gaze points and 3D fixations using the libraries. The Gaze3DFix toolkit, including both libraries and the graphical user interface, is available as open-source software at https://github.com/applied-cognition-research/Gaze3DFix .

EEG Negativity in Fixations Used for Gaze-Based Control: Toward Converting Intentions into Actions with an Eye-Brain-Computer Interface.

We usually look at an object when we are going to manipulate it. Thus, eye tracking can be used to communicate intended actions. An effective human-machine interface, however, should be able to differentiate intentional and spontaneous eye movements. We report an electroencephalogram (EEG) marker that differentiates gaze fixations used for control from spontaneous fixations involved in visual exploration. Eight healthy participants played a game with their eye movements only. Their gaze-synchronized EEG data (fixation-related potentials, FRPs) were collected during game's control-on and control-off conditions. A slow negative wave with a maximum in the parietooccipital region was present in each participant's averaged FRPs in the control-on conditions and was absent or had much lower amplitude in the control-off condition. This wave was similar but not identical to stimulus-preceding negativity, a slow negative wave that can be observed during feedback expectation. Classification of intentional vs. spontaneous fixations was based on amplitude features from 13 EEG channels using 300 ms length segments free from electrooculogram contamination (200-500 ms relative to the fixation onset). For the first fixations in the fixation triplets required to make moves in the game, classified against control-off data, a committee of greedy classifiers provided 0.90 ± 0.07 specificity and 0.38 ± 0.14 sensitivity. Similar (slightly lower) results were obtained for the shrinkage Linear Discriminate Analysis (LDA) classifier. The second and third fixations in the triplets were classified at lower rate. We expect that, with improved feature sets and classifiers, a hybrid dwell-based Eye-Brain-Computer Interface (EBCI) can be built using the FRP difference between the intended and spontaneous fixations. If this direction of BCI development will be successful, such a multimodal interface may improve the fluency of interaction and can possibly become the basis for a new input device for paralyzed and healthy users, the EBCI "Wish Mouse."

Dynamic Causal Modeling of Hippocampal Links within the Human Default Mode Network: Lateralization and Computational Stability of Effective Connections.

The purpose of this paper was to study causal relationships between left and right hippocampal regions (LHIP and RHIP, respectively) within the default mode network (DMN) as represented by its key structures: the medial prefrontal cortex (MPFC), posterior cingulate cortex (PCC), and the inferior parietal cortex of left (LIPC) and right (RIPC) hemispheres. Furthermore, we were interested in testing the stability of the connectivity patterns when adding or deleting regions of interest. The functional magnetic resonance imaging (fMRI) data from a group of 30 healthy right-handed subjects in the resting state were collected and a connectivity analysis was performed. To model the effective connectivity, we used the spectral Dynamic Causal Modeling (DCM). Three DCM analyses were completed. Two of them modeled interaction between five nodes that included four DMN key structures in addition to either LHIP or RHIP. The last DCM analysis modeled interactions between four nodes whereby one of the main DMN structures, PCC, was excluded from the analysis. The results of all DCM analyses indicated a high level of stability in the computational method: those parts of the winning models that included the key DMN structures demonstrated causal relations known from recent research. However, we discovered new results as well. First of all, we found a pronounced asymmetry in LHIP and RHIP connections. LHIP demonstrated a high involvement of DMN activity with preponderant information outflow to all other DMN regions. Causal interactions of LHIP were bidirectional only in the case of LIPC. On the contrary, RHIP was primarily affected by inputs from LIPC, RIPC, and LHIP without influencing these or other DMN key structures. For the first time, an inhibitory link was found from MPFC to LIPC, which may indicate the subjects' effort to maintain a resting state. Functional connectivity data echoed these results, though they also showed links not reflected in the patterns of effective connectivity. We suggest that such lateralized architecture of hippocampal connections may be related to lateralization phenomena in verbal and spatial domains documented in human neurophysiology, neuropsychology, and neurolinguistics.

Changes in snoRNA and snRNA Abundance in the Human, Chimpanzee, Macaque, and Mouse Brain.

Small nuclear and nucleolar RNAs (snRNAs and snoRNAs) are known to be functionally and evolutionarily conserved elements of transcript processing machinery. Here, we investigated the expression evolution of snRNAs and snoRNAs by measuring their abundance in the frontal cortex of humans, chimpanzees, rhesus monkeys, and mice. Although snRNA expression is largely conserved, 44% of the 185 measured snoRNA and 40% of the 134 snoRNA families showed significant expression divergence among species. The snRNA and snoRNA expression divergence included drastic changes unique to humans: A 10-fold elevated expression ofU1snRNA and a 1,000-fold drop in expression ofSNORA29 The decreased expression ofSNORA29might be due to two mutations that affect secondary structure stability. Using in situ hybridization, we further localizedSNORA29expression to nucleolar regions of neuronal cells. Our study presents the first observation of snoRNA abundance changes specific to the human lineage and suggests a possible mechanism underlying these changes.

Effective Connectivity within the Default Mode Network: Dynamic Causal Modeling of Resting-State fMRI Data.

The Default Mode Network (DMN) is a brain system that mediates internal modes of cognitive activity, showing higher neural activation when one is at rest. Nowadays, there is a lot of interest in assessing functional interactions between its key regions, but in the majority of studies only association of Blood-oxygen-level dependent (BOLD) activation patterns is measured, so it is impossible to identify causal influences. There are some studies of causal interactions (i.e., effective connectivity), however often with inconsistent results. The aim of the current work is to find a stable pattern of connectivity between four DMN key regions: the medial prefrontal cortex (mPFC), the posterior cingulate cortex (PCC), left and right intraparietal cortex (LIPC and RIPC). For this purpose functional magnetic resonance imaging (fMRI) data from 30 healthy subjects (1000 time points from each one) was acquired and spectral dynamic causal modeling (DCM) on a resting-state fMRI data was performed. The endogenous brain fluctuations were explicitly modeled by Discrete Cosine Set at the low frequency band of 0.0078-0.1 Hz. The best model at the group level is the one where connections from both bilateral IPC to mPFC and PCC are significant and symmetrical in strength (p < 0.05). Connections between mPFC and PCC are bidirectional, significant in the group and weaker than connections originating from bilateral IPC. In general, all connections from LIPC/RIPC to other DMN regions are much stronger. One can assume that these regions have a driving role within the DMN. Our results replicate some data from earlier works on effective connectivity within the DMN as well as provide new insights on internal DMN relationships and brain's functioning at resting state.

Effects of driving time on microsaccadic dynamics.

Driver fatigue is a common cause of car accidents. Thus, the objective detection of driver fatigue is a first step toward the effective management of fatigue-related traffic accidents. Here, we investigated the effects of driving time, a common inducer of driver fatigue, on the dynamics of fixational eye movements. Participants drove for 2 h in a virtual driving environment while we recorded their eye movements. Microsaccade velocities decreased with driving time, suggesting a potential effect of fatigue on microsaccades during driving.

Attentional dynamics during free picture viewing: Evidence from oculomotor behavior and electrocortical activity.

Most empirical evidence on attentional control is based on brief presentations of rather abstract stimuli. Results revealed indications for a dynamic interplay between bottom-up and top-down attentional mechanisms. Here we used a more naturalistic task to examine temporal signatures of attentional mechanisms on fine and coarse time scales. Subjects had to inspect digitized copies of 60 paintings, each shown for 40 s. We simultaneously measured oculomotor behavior and electrophysiological correlates of brain activity to compare early and late intervals (1) of inspection time of each picture (picture viewing) and (2) of the full experiment (time on task). For picture viewing, we found an increase in fixation duration and a decrease of saccadic amplitude while these parameters did not change with time on task. Furthermore, early in picture viewing we observed higher spatial and temporal similarity of gaze behavior. Analyzing electrical brain activity revealed changes in three components (C1, N1 and P2) of the eye fixation-related potential (EFRP); during picture viewing; no variation was obtained for the power in the frontal beta- and in the theta activity. Time on task analyses demonstrated no effects on the EFRP amplitudes but an increase of power in the frontal theta and beta band activity. Thus, behavioral and electrophysiological measures similarly show characteristic changes during picture viewing, indicating a shifting balance of its underlying (bottom-up and top-down) attentional mechanisms. Time on task also modulated top-down attention but probably represents a different attentional mechanism.

I see you remembering: what eye movements can reveal about process characteristics of prospective memory.

Prospective memory performance describes the delayed execution of an intended action. As this requires a mixture of memory and attentional control functions, current research aims at delineating the specific processes associated with solving a prospective memory task. Therefore, the current study measured, analysed and compared eye movements of participants who performed a prospective memory, a free viewing, and a visual search task. By keeping constant the prospective memory cue as well as the context of tasks, we aimed at putting the processes of solving prospective memory tasks into context. The results show, that when a prospective memory task is missed, the continuous gaze behaviour is rather similar to the gaze behaviour during free viewing. When the prospective memory task is successfully solved, on the other hand, average gaze behaviour is between free viewing and visual search. Furthermore, individual differences in eye movements were found between low and high performers. Our data suggest that a prospective memory task can be solved in different ways, therefore different processes can be observed.

Gaze transfer in remote cooperation: is it always helpful to see what your partner is attending to?

Establishing common ground in remote cooperation is challenging because nonverbal means of ambiguity resolution are limited. In such settings, information about a partner's gaze can support cooperative performance, but it is not yet clear whether and to what extent the abundance of information reflected in gaze comes at a cost. Specifically, in tasks that mainly rely on spatial referencing, gaze transfer might be distracting and leave the partner uncertain about the meaning of the gaze cursor. To examine this question, we let pairs of participants perform a joint puzzle task. One partner knew the solution and instructed the other partner's actions by (1) gaze, (2) speech, (3) gaze and speech, or (4) mouse and speech. Based on these instructions, the acting partner moved the pieces under conditions of high or low autonomy. Performance was better when using either gaze or mouse transfer compared to speech alone. However, in contrast to the mouse, gaze transfer induced uncertainty, evidenced in delayed responses to the cursor. Also, participants tried to resolve ambiguities by engaging in more verbal effort, formulating more explicit object descriptions and fewer deictic references. Thus, gaze transfer seems to increase uncertainty and ambiguity, thereby complicating grounding in this spatial referencing task. The results highlight the importance of closely examining task characteristics when considering gaze transfer as a means of support.

Fixation based event-related fmri analysis: using eye fixations as events in functional magnetic resonance imaging to reveal cortical processing during the free exploration of visual images.

Eye movements, comprising predominantly fixations and saccades, are known to reveal information about perception and cognition, and they provide an explicit measure of attention. Nevertheless, fixations have not been considered as events in the analyses of data obtained during functional magnetic resonance imaging (fMRI) experiments. Most likely, this is due to their brevity and statistical properties. Despite these limitations, we used fixations as events to model brain activation in a free viewing experiment with standard fMRI scanning parameters. First, we found that fixations on different objects in different task contexts resulted in distinct cortical patterns of activation. Second, using multivariate pattern analysis, we showed that the BOLD signal revealed meaningful information about the task context of individual fixations and about the object being inspected during these fixations. We conclude that fixation-based event-related (FIBER) fMRI analysis creates new pathways for studying human brain function by enabling researchers to explore natural viewing behavior.

Comparing two types of engineering visualizations: task-related manipulations matter.

This study focuses on the comparison of traditional engineering drawings with a CAD (computer aided design) visualization in terms of user performance and eye movements in an applied context. Twenty-five students of mechanical engineering completed search tasks for measures in two distinct depictions of a car engine component (engineering drawing vs. CAD model). Besides spatial dimensionality, the display types most notably differed in terms of information layout, access and interaction options. The CAD visualization yielded better performance, if users directly manipulated the object, but was inferior, if employed in a conventional static manner, i.e. inspecting only predefined views. An additional eye movement analysis revealed longer fixation durations and a stronger increase of task-relevant fixations over time when interacting with the CAD visualization. This suggests a more focused extraction and filtering of information. We conclude that the three-dimensional CAD visualization can be advantageous if its ability to manipulate is used.

Saccadic context indicates information processing within visual fixations: evidence from event-related potentials and eye-movements analysis of the distractor effect.

Attention, visual information processing, and oculomotor control are integrated functions of closely related brain mechanisms. Recently, it was shown that the processing of visual distractors appearing during a fixation is modulated by the amplitude of its preceding saccade (Pannasch & Velichkovsky, 2009). So far, this was demonstrated only at the behavioral level in terms of saccadic inhibition. The present study investigated distractor-related brain activity with cortical eye fixation-related potentials (EFRPs). Moreover, the following saccade was included as an additional classification criterion. Eye movements and EFRPs were recorded during free visual exploration of paintings. During some of the fixations, a visual distractor was shown as an annulus around the fixation position, 100 ms after the fixation onset. The saccadic context of a fixation was classified by its preceding and following saccade amplitudes with the cut-off criterion set to 4° of visual angle. The prolongation of fixation duration induced by distractors was largest for fixations preceded and followed by short saccades. EFRP data revealed a difference in distractor-related P2 amplitude between the saccadic context conditions, following the same trend as in eye movements. Furthermore, influences of the following saccade amplitude on the latency of the saccadic inhibition and on the N1 amplitude were found. The EFRP results cannot be explained by the influence of saccades per se since this bias was removed by subtracting the baseline from the distractor EFRP. Rather, the data suggest that saccadic context indicates differences in how information is processed within single visual fixations.

On the control of visual fixation durations in free viewing of complex images.

The mechanisms for the substantial variation in the durations of visual fixations in scene perception are not yet well understood. During free viewing of paintings, gaze-contingent irrelevant distractors (Exp. 1) and non-gaze-related time-locked display changes (Exp. 2) were presented. We demonstrated that any visual change-its onset and offset-prolongs the ongoing fixation (i.e., delays the following saccade), strongly suggesting that fixation durations are under the direct control of the stimulus information. The strongest influence of distraction was observed for fixations preceded by saccades within the parafoveal range (<5° of visual angle). We assume that these fixations contribute to the focal in contrast to the ambient mode of attention (Pannasch & Velichkovsky, Visual Cognition, 17, 1109-1131, 2009; Velichkovsky, Memory, 10, 405-419, 2002). Recent findings about two distinct "subpopulations of fixations," one under the direct and another under the indirect control of stimulation (e.g., Henderson & Smith, Visual Cognition, 17, 1055-1082, 2009), are reconsidered in view of these results.

Saccadic peak velocity sensitivity to variations in mental workload.

INTRODUCTION: For research and applications in the field of (neuro)ergonomics, it is of increasing importance to have reliable methods for measuring mental workload. In the present study we examined the hypothesis that saccadic eye movements can be used for an online assessment of mental workload. METHODS: Saccadic main sequence (amplitude, duration and peak velocity) was used as a diagnostic measure of mental workload in a virtual driving task with three complexity levels. We tested 18 drivers in the SIRCA driving simulator while their eye movements were recorded. The Wickens' multiple resources model was used as theoretical framework. Changes in mental workload between the complexity levels were evaluated multidimensionally, using subjective rating, performance in a secondary task, and other behavioral indices. RESULTS: Saccadic peak velocity decreased (7.2 visual degrees/s) as the mental workload increased, as measured by scores of mental workload test (15.2 scores) and the increase of the reaction time on the secondary task (46 ms). DISCUSSION: Saccadic peak velocity is affected by variations in mental workload during ecologically valid tasks. We conclude that saccadic peak velocity could be a useful diagnostic index for the assessment of operators' mental workload and attentional state in hazardous environments.

Yarbus, eye movements, and vision.

The impact of Yarbus's research on eye movements was enormous following the translation of his book Eye Movements and Vision into English in 1967. In stark contrast, the published material in English concerning his life is scant. We provide a brief biography of Yarbus and assess his impact on contemporary approaches to research on eye movements. While early interest in his work focused on his study of stabilised retinal images, more recently this has been replaced with interest in his work on the cognitive influences on scanning patterns. We extended his experiment on the effect of instructions on viewing a picture using a portrait of Yarbus rather than a painting. The results obtained broadly supported those found by Yarbus.